Yes, Topology Matters in Decentralized Optimization: Refined Convergence and Topology Learning under Heterogeneous Data

One of the key challenges in federated and decentralized learning is to design algorithms that efficiently deal with highly heterogeneous data distributions across agents. In this paper, we revisit the analysis of Decentralized Stochastic Gradient Descent algorithm (D-SGD), a popular decentralized learning algorithm, under data heterogeneity. We exhibit the key role played by a new quantity, that we call neighborhood heterogeneity, on the convergence rate of D-SGD. Unlike prior work, neighborhood heterogeneity is measured at the level of the neighborhood of an agent in the graph topology. By coupling the topology and the heterogeneity of the agents' distributions, our analysis sheds light on the poorly understood interplay between these two concepts in decentralized learning. We then argue that neighborhood heterogeneity provides a natural criterion to learn sparse data-dependent topologies that reduce (and can even eliminate) the otherwise detrimental effect of data heterogeneity on the convergence time of D-SGD. For the important case of classification with label skew, we formulate the problem of learning such a good topology as a tractable optimization problem that we solve with a Frank-Wolfe algorithm. Our approach provides a principled way to design a sparse topology that balances the number of iterations and the per-iteration communication costs of D-SGD under data heterogeneity

Reachability of Five Gossip Protocols

Gossip protocols use point-to-point communication to spread information within a network until every agent knows everything. Each agent starts with her own piece of information (‘secret’) and in each call two agents will exchange all secrets they currently know. Depending on the protocol, this leads to different distributions of secrets among the agents during its execution. We investigate which distributions of secrets are reachable when using several distributed epistemic gossip protocols from the literature. Surprisingly, a protocol may reach the distribution where all agents know all secrets, but not all other distributions. The five protocols we consider are called 햠햭햸, 햫햭햲, 햢햮, 햳햮햪, and 햲햯햨. We find that 햳햮햪 and 햠햭햸 reach the same distributions but all other protocols reach different sets of distributions, with some inclusions. Additionally, we show that all distributions are subreachable with all five protocols: any distribution can be reached, if there are enough additional agents

A Decentralized Approach to Network-Aware Service Composition

Dynamic service composition represents a key feature for service-based applications operating in dynamic and large scale network environments, as it allows leveraging the variety of offered services, and to cope with their volatility. However, the high number of services and the lack of central control pose a significant challenge for the scalability and effectiveness of the composition process. We address this problem by proposing a fully decentralized approach to servicecomposition, based on the use of a gossip protocol to support information dissemination and decision making. The proposed system builds and maintains acomposition of services that fulfills both functional and non functional requirements. For the latter, we focus in particular on requirements concerning the composite service completion time, taking into account both the response time and the impact of network latency. Simulation experiments show that our solution converges quickly to a feasible composition and can self-adapt to dynamic changes concerning both service availability and network latency

Aggregation techniques for the internet of things: An overview

Internet of Thing (IoT) can be generally defined as a network connecting millions of smart objects, most of them equipped with sensors. Since sensors are devices generating a huge amount of data, the transmission of raw data to the edge nodes and then to higher level cloud nodes may give rise to transmission delays and energy consumption. Furthermore, sensors are characterized by limited resources. For all these reasons, aggregation techniques are required to reduce the size of data to be transmitted and stored, while maintaining a reasonable level of approximation. In this paper, we propose an overview of a set of aggregation techniques which may be exploited in IoT. We present a set of techniques, ranging from Space Filling Curves, to Q-digest, Wavelets, Gossip aggregation, and Compressive Sensing. We also show how these techniques are exploited in IoT applications